On-Time Versus Late Story Completion Process for MESOS
Introduction
This report compares the processes used by the project when it delivers user stories on-time versus when late.
Software development and maintenance projects that use an agile life-cycle, such as Scrum, document their requirements as ‘User Stories’. These stories are intended to describe a small software feature that is:
1. Understandable to the customer/user,
2. Can be implemented during one sprint (time-boxed delivery period), and
3. Can be tested or demonstrated.
User stories that have not yet been implemented are maintained in a list called a ‘backlog’ and are subject to change as the customer/user needs change.
An active user story is a user story that is being implemented during the currently active sprint. Once a user story becomes active, it becomes a promise from the software developer to the customer/user to implement the user story and demonstrate it at the end of the sprint.
When a promise (active user story) cannot be delivered during the current sprint and the customer/user still wants the user story to be delivered, then the customer may view the incomplete user story as late. (The project may view the user story as a promise being kept late.)
This analysis report compares the process(es) that the project follows when it delivers on time versus the process(es) the project follows when it is significantly late. Being significantly late is a strong but not an absolute indicator that the process being used is different than the process(es) used for stories delivered on-time. For this report, the lateness threshold used in the analysis was set so that a story exceeding this delivery time has a 90% chance that its delivery indicated a real difference in some aspect of the project’s process.
The processes associated with stories delivered on-time as planned are labeled as the standard processes. For this report, on-time delivery is taken as expected typical behavior for the project.
The processes associated with stories delivered late (not as promised) are labeled as the exception processes. For this report, late delivery is taken as unusual behavior to be avoided by the project.
Process Analysis
In comparing the processes for standard versus exception behavior, consider:
If the standard process implementations are much shorter than the exception process implementations, review the steps that are in the exception process to determine if the steps are needed and why they are occurring. Performing a root cause analysis on the trigger conditions associated with these steps may assist in eliminating some exception cases.
If the standard process implementations are much longer than the exception process implementations, review the steps in the standard process to determine how the steps are assisting the project in making on-time deliveries. This situation can be difficult to understand, since performing additional steps is leading to better project delivery times. Performing a root cause analysis, from a positive “opportunity” perspective, about the trigger conditions associated with these steps may assist in understanding how the steps help the project.
If the standard and exception process implementations largely contain the same steps, then the differences are likely in the underlying process parameters being changed at each process step. For example, both types of process implementation might assign a task with the assignment to a junior versus senior developer as the difference in the process parameter. Analyzing these differences is challenging, since the processes are the same except for details of the implementation. Information to compare these parameter settings is found in the section “Process Setting Comparison”.
Comparison of Minimum-Length Process Implementations
The following analysis compares the minimum length process implementations for the project’s standard behavior—that is, when the project makes an on-time delivery—versus the project’s exceptional behavior—that is, when the project is extraordinarily late (as defined above at the 90% threshold). The minimum-length process is the shortest number of steps the process used across all actual process implementations when the project ultimately made a delivery. The minimum length workflow diagrams are show in the “Minimum Length Process Implementation Workflow Diagrams” section below. A computed comparison of the distance between each standard and exception minimum-length process is shown in the table at the end of this section.
See the section below on “Process Calculation Methodology” to understand why multiple processes may be shown.
Minimum Length Process Implementation Workflow Diagrams
Standard Process(es)
Exception Process(es)
Computed Distances Between Minimum Length Standard and Exception Processes
(0 ≤ Distance ≤ 1, 0 means identical, 1 means completely different)
| Exception Process #1 | Exception Process #2 | Exception Process #3 | Exception Process #4 | |
|---|---|---|---|---|
| Standard Process # 1 | 1 | 1 | 1 | 0.4269231 |
Comparison of Average Process Implementations
The following analysis compares the average process implementations for the project’s standard behavior—that is, when the project makes an on-time delivery—versus the project’s exceptional behavior—that is, when the project is extraordinarily late (as defined above at the 90% threshold). The average process is computed used across all actual process implementations when the project ultimately made a delivery using a clustering algorithm to determine an “average” that has selected properties. See the section on “Determination of the ‘Average’ Process” for a discussion on the clustering algorithm used. (Other clustering algorithms with different properties may also be selected.) The average workflow diagrams are show in the “Average Process Implementation Workflow Diagrams” section below. A computed comparison of the distance between each standard and exception average process is shown in the table at the end of this section.
See the section below on “Process Calculation Methodology” to understand why multiple processes may be shown.
Average Process Implementation Workflow Diagrams
Standard Process(es)
Exception Process(es)
Computed Distances Between Average Standard and Exception Processes
(0 ≤ Distance ≤ 1, 0 means identical, 1 means completely different)
| Exception Process #1 | |
|---|---|
| Standard Process # 1 | 0.5434524 |
Comparison of Maximum-Length Process Implementations
The following analysis compares the maximum length process implementations for the project’s standard behavior—that is, when the project makes an on-time delivery—versus the project’s exceptional behavior—that is, when the project is extraordinarily late (as defined above at the 90% threshold). The maximum-length process is the longest number of steps the process used across all actual process implementations when the project ultimately made a delivery. The maximum length workflow diagrams are show in the “Maximum Length Process Implementation Workflow Diagrams” section below. A computed comparison of the distance between each standard and exception maximum-length process is shown in the table at the end of this section.
See the section below on “Process Calculation Methodology” to understand why multiple processes may be shown.
Maximum Length Process Implementation Workflow Diagrams
Standard Process(es)
Exception Process(es)
Computed Distances Between Maximum Length Standard and Exception Processes
(0 ≤ Distance ≤ 1, 0 means identical, 1 means completely different)
| Exception Process #1 | Exception Process #2 | |
|---|---|---|
| Standard Process # 1 | 0.2552296 | 0.5139456 |
Process Setting Comparison
If the minimum distance between the standard and exception process implementations for minimum length, average, or maximum length are less than 0.1, then differences in performance between standard and exception process implementations may not be generated by different structures of the process implementations but rather by the setting value differences (for example, changing user story sizes, where the exception process implementations may be driven by incorrect settings for story size) between the two.
In this case, the minimum distance was 0.255229591836735%. This means that the process implementation differences are probably not influenced by setting value differences.
The following heat map graphs provided a basis for comparing the parameter setting differences between the standard and exception process differences. The standard process parameters are shown on the left and the exception process parameters on the right. In each heat map, the x-axis value is the value the parameter is being changed from and the y-axis is the value it is being changed to. Colors indicate the frequency of the from-to pair, with lighter colors showing more frequent changes from the corresponding “from” value to the “to” value.
For each parameter, analyze the patterns shown by the standard versus exception process implementation to detect a possible root cause for on-time delivery differences between the standard and exception process implementations.
[[1]] 
[[2]] 
[[3]] 
[[4]] 
[[5]] 
[[6]] 
[[7]] 
[[8]] 
[[9]] 
[[10]] 
[[11]] 
[[12]] 
[[13]] 
[[14]] 
[[15]] 
[[16]] 
[[17]] 
[[18]] [[19]] 
[[20]] 
[[21]] 
[[22]] 
[[23]] 
[[24]] 
[[25]] 
[[26]] 
[[27]] 
[[28]] 
[[29]] 
[[30]] 
[[31]] 
[[32]] 
[[33]] 
[[34]] 
[[35]] [[36]] [[37]] 
[[38]] 
[[39]] 
[[40]] 
[[41]] 
[[42]] 
[[43]] 
[[44]] 
[[45]] 
[[46]] 
[[47]] 
[[48]] 
[[49]] 
[[50]] 
[[51]] 
[[52]] 
Although the standard process implementations use the Version parameter, the exception process implementations do not. Therefore no comparison heat map is displayed.
Although the standard process implementations use the External issue URL parameter, the exception process implementations do not. Therefore no comparison heat map is displayed.
Although the standard process implementations use the Epic Child parameter, the exception process implementations do not. Therefore no comparison heat map is displayed.
Although the standard process implementations use the duedate parameter, the exception process implementations do not. Therefore no comparison heat map is displayed.
Although the standard process implementations use the External issue ID parameter, the exception process implementations do not. Therefore no comparison heat map is displayed.
Although the standard process implementations use the environment parameter, the exception process implementations do not. Therefore no comparison heat map is displayed.
Although the exception process implementations use the Version parameter, the standard process implementations do not. Therefore no comparison heat map is displayed.
Although the exception process implementations use the External issue URL parameter, the standard process implementations do not. Therefore no comparison heat map is displayed.
Although the exception process implementations use the Epic Child parameter, the standard process implementations do not. Therefore no comparison heat map is displayed.
Although the exception process implementations use the duedate parameter, the standard process implementations do not. Therefore no comparison heat map is displayed.
Although the exception process implementations use the External issue ID parameter, the standard process implementations do not. Therefore no comparison heat map is displayed.
Although the exception process implementations use the environment parameter, the standard process implementations do not. Therefore no comparison heat map is displayed.
Process Calculation Methodology
Identification of Multiple Process Variations: Minimum, Average, and Maximum
This report does not identify the project’s singular process. Instead, this report identifies a set of processes depending on the workflow used by the project when it delivered stories on time versus late. For both the on-time and late processes, the report lists:
- The actual process implementation(s) that achieved an on-time or significantly late delivery in the minimum number of steps. (One or more actual process implementations may have the same number of minimum steps. However, no other process implementation had fewer steps than the implementations shown.)
- A calculated process that represents the average process used to achieve an on-time or significantly late deliver.
- The actual process implementations that achieved an on-time or significantly late delivery in the maximum number of steps. (One or more actual process implementations may have the same number of maximum steps. However, no other process implementation had more steps than the implementations shown.)
Personnel who define processes often refer to the project process as if the project performed in exactly the same way every time a particular work action is taken. (This is called “idealizing” a process.) Furthermore, many projects have a documented process. However, regardless of whether the process is documented or not or how repetitious the process might be, in practice, every time a process is performed there are differences in that particular implementation versus any other. There may be differences in timing, in the exact sequence of some steps, or in the personnel performing a particular step. There is no true single process. Some of the process differences are significant. Others are not.
For example, consider the process you follow when you get dressed. If, in one process implementation, you put on your left shoe first and then your right shoe, then this is clearly a different process implementation than if you put your right shoe on first followed by your left–but the details of the two process implementations have no material impact on achieving the state of “being dressed”. Many people would therefore describe these processes as being the same. In contrast, most people would describe the process of putting on a sock before putting on a shoe as significantly different from putting on the shoe and then the sock with respect to achieving the state of “being dressed”.
Although this report can identify differences in a process, it can only suggest but not confirm whether the differences are material with respect to on-time or significantly late user story completion. The report reader will need to judge the importance of these differences.
Determination of the “Average” Process
Given that work is performed on a project in many different ways that all achieve on-time delivery of active user stories, there is no ultimate single process that describes how work is done. However, we can calculate what the project’s typical process. What we mean by the “typical” process can also have different definitions, just like what is meant by the “middle” of a set of can be described by the arithmetic mean or the median. (For example, for the set of numbers 1, 3, 3, 4, 5, 6, the mean is 3.6666667 and the median is 3.5, while for the numbers 1, 2, 3, 4, 5, 6, the mean and median are both 3.5.)
For the purposes of determining the project’s average process, the calculation is done using a clustering algorithm. The specific clustering algorithm used in this case. See “Appendix A: Computation Technical Details” for a link with more details on the clustering method.
The clustering algorithm can yield a single representative “average” process, two representative average processes, and so on, up to the number of different process implementations actually observed. Selecting a larger number of representative processes yields a lower total error rate across all processes, but correspondingly diminishes the explanatory power of the clustering results. There is no universally agreed algorithm for determining the balance between decreasing total error rate versus explanatory power; determining the right balance is currently based on the preferences of this report’s user. The number of representative average reports selected for this particular report was 1. See “Appendix B: Cluster Dendrogram Analysis” for a discussion and graphical depiction of the balance between total error rate and number of representative average processes.
Appendices
Appendix A: Computation Technical Details
The method used to calculate distances between processes is the Jaro-Winkler distance with penalty factor p = 0.1. See https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance.
The clustering method used to calculate the “average” process is Ward’s method. See https://en.wikipedia.org/wiki/Ward%27s_method.
Appendix B: Cluster Dendrogram Analysis
A dendrogram shows how the clustering algorithm was applied to create the “average” process shown in the results. At the bottom of the dendrogram, the clustering algorithm begins be treating all of the observed process implementations as separate. Each label along the x-axis represents a process, though the labels may not be readable. (The labels are references to particular process implementations which can be retrieved from the Nymbul Mineral© tool if desired.) As the clustering algorithm computes the distance between the process implementation as described in Appendix A, it joins the two most similar process implementations into a single cluster. This joining, or clustering (hence the name), groups together process implementations or previously clustered process implementations based on their calculate distance from each other. Each clustering step, moving from the bottom of the dendrogram to the top, results in fewer process implementations representing all observations until only a single “average” process implementation remains.
The vertical distance, as shown on the y-axis, in the dendrogram from joining one process cluster to the next shows how different the process implementation clusters are. Larger vertical distances before a joining represent greater differences and small vertical distances represent small differences.
Drawing a horizontal line across the dendrogram and selecting the clusters that are next below the horizontal line provides the means to choose a set of clusters that represents the overall set of process implementations at a particular level of “difference” based on cluster dissimilarity (or similarity) based on the vertical distance. Choosing only one process to represent the entire set of process implementation is equivalent to putting a horizontal line at the very top of the dendrogram. Someone using this report might choose to place such a horizontal line lower in the dendrogram, thereby having more than a single process implementation representing the entire set. The determination of where to place the line would be based on:
Choosing an acceptable (to the user) distance between process implementation clusters as measured based on dendrogram vertical distances.
Choosing a useful (to the user) number of process implementation clusters.
Note that the total number of process implementations in other the standard or exception dendrograms, as shown by the number of labels at the bottom of each, does not indicate anything except the frequency that the project engages in its “standard” versus “exception” (or non-standard) process. From most projects, we would expect to see more standard process implementations than exception implementations. (Projects with more exception implementations than standard should examine how the project is run, since this behavior indicates significant project instability.)
Standard Process Dendrogram Diagram
Exception Process Dendrogram Diagram
